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A general evaluation design was developed to examine 
the effectiveness of a computer-based, multimedia simulation test on 
California smog check mechanics. The simulation test operated on an 
Apple Macintosh Ilci, with a single touchscreen color monitor 
controlling a videodisc player; it had three parts: 
introduction-tutorial-help, data, and test. It was compared to a 
hands.-on, actual vehicle test. The two tests were as equivalent as 
possible. The hands-on test was essentially a printout of the nine 
test items and specific questions from the simulation test. The 
evaluation gathered two kinds of data: test performance and 
subjective responses to the test experience. Currently certified smog 
check mechanics were recruited and paid to participate in the study. 
This sample was randomized between the two testing methods; 38 were 
tested in each group. A Rasch analysis was chosen for a reliability 
study of the test. The high item separation reliability indicated 
that, in general, persons responded consistently to the test 
according to their performance capabilities. The comparison of the 
performance between the simulation test group and the hands-on test 
group indicated the difference was not statistically significant. 
Data showed that the probability was very high that the differences 
between the group means were the result of chance. (Appendixes which 
comprise at least half of the document include the following: 
background and simulation test documents, mechanics survey, sample 
selection documents, raw data score sheets, and test proctors' 
observations/recommendations .). (YLB) 
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EXECUTIVE SUBOIARY 



INTRODUCTION AND BACKGROUND 

In January of 1988, a B.A.R. -funded study, Hands-On Verification Of 
Mechanics Training: A Cost-Effectiveness S tiidv Of Videodisc Simulation 
(Maher, 1988), recommended to the Bureau of Automotive Repair (B.A.R.) 
that: 

1. B.A.R. should no longer actually conduct mechanic training, but 
should continue to test, certify and license. 

2. B.A.R. should test, certify and license mechanics through 
comprehensive "hands-on" simulation testing. 

The California State Polytechnic University at Pomona (Cal Poly) received two 
subsequent contracts from B.A.R., parts of which were designated to 
implement a demonstration project of computer-based, multimedia 
simulation testing. 

This document reports on the experimental evaluation trals of the prototype 
simulation test, as compared to a "hands-on" test using actual vehicles. 

GENERAL DESCRIPTION OF THE SIMULATION TEST 

The simulation test operates on an Apple Macintosh Ilci, with a single 
touchscreen color monitor, controlling a videodisc player. The primary 
software controlling the simulation is MacroMedia Director ver 3.1. 

The simulation test has three parts. Introduction-Tutorial- Help Section; This 
section orients the mechanic to the operation of the display device by 
introducing the computer and explaining the methods of both presentation 
and appropriate mechanic response. TTiis "Tutorial" section is liien used as 
the "Help" section. Data Section: The data section of the test is unavailable 
to the mechanic. It contains the mechanics' logon information, including 
B.A.R. certificate number, the touchscreen/keyboard entry responses of the 
mechanic to test questions, etc. Test Section: The essential framework of 
this simulation examination is a smog check, including a visual inspection, 
functional check, and diagnosis of emission problems. Three vehicles were 
used. The actual test consists of nine test "items" divided into a total of 44 
individual questions. Questions are fully simulated through audio/video/ 
graphic displays. 

EVALUATION DESIGN 

The general evaluation design was developed to examine the effectiveness of a 
computer-based, multimedia simulation test on California smog check 
mechanics. The simulation test was compared to a "hands-on", actu£d vehicle 
test. The two tests were as equivalent as possible. The "hands-on" test itself 



was essentially a "printout" of the nine test items and specific questions from 
the simulation test. The evaluation gathered two kinds of data,, test 
performance and subjective responses to the test experience. 

Currently certified smog check mechanics were recruited- and paid to 
participate in the study. This sample was randomized between the two 
testing methods. 

The purpose of the evaluation was the comparison of those two testing 
methodologies, not validating either test for use with the entire population. 
In the current case both groups are considered the same. The question of 
interest is whether there is no significant difference in performance between 
the groups, suggesting that the two methods of examination are essentially 
equ^, £md yield equal results. 

RESULTS AND ANALYSIS 

To properly evaluate the performEmce of the two groups on the simulation 
examination £md the '*h£mds-on" version, the test itself had to be anal3^d for 
reliability. A Rasch analysis was chosen for the reliability study. The final 
calibration was made after eliminating "misfitting" items. 



Original Calibration: 


Person Separation Reliability 


=.52 




Item Separation (Test) Reliability 


=.82 


Final Calibration: 


Person Separation Reliability 


=.60 




Item Separation (Test) Reliability 


=.86 



The high item separation reliability, or test reliability, indicated that, in 
general, persons responded consistently to the test according to their 
performance capabilities. 



Once the test reliability was determined, all items and all scores were used in 
the comparison of the performance between the simulation test group and 
the "hands-on" test group. The summary statistics for the raw scores are 
shown below: 



RAW SCORES 








Number Mean 


Standard Deviation 


Simulation Test 


38 44.184 


3.525 


"Hands-On" Test 


38 43.737 


3.117 


t= .586 


Dei^ees of Freedom = 74 


Probability = .560 



The difference between the two groups is not statistically significant. These 
data show that the probability is in fact very high that the differences between 
the group means is the result of chance. 
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The summary statistics for the survey averages are shown below: 



Number 



Mean 



Standard Deviation 



Simulation Test 
-Hands-On" Test 



t = 1.056 



10 2.258 .443 

10 2.050 .437 

Degrees of Freedom =18 Probability = .305 



These statistics show that the differences between the two groups is not 
significant. 



The purpose of the B.A.R. Demonstration Project was to examine the 
feasibility of creating a computer-based multimedia simulation examination 
that could provide B.A.R with the same level of confidence in assessing the 
ability level of mechanics as a "hands-on" actual vehicle examination, but in a 
more cost-effective manner. The statistical comparison of the certified smog 
check mechanics that were examined with the prototype simulation test as 
compared with an equivalent "hands-on" test indicate that the simulation test 
is not significantly different from the "hands-on" test. 

The survey results indicated that there was no significant difference between 
the opinions of the simulation and "hands-on" groups concerning their 
respective testing experiences. The subjective reaction of the simulation 
group to their experience does not seem to contain any issues that would 
inhibit the acceptance of simulation testing as a new test methodology for 
B.A.R. These findings are congruent with, and supported by, the general 
literature on simulation training and testing (Maher, 1988). 

GenaxdRecommendatiam: 

1. BJLR. should convert its current written examination to a computer-based 
multimedia simulation examination, using the current demonstration design 
documents and simulation test, with at least the modifications discussed 
below, as the working prototype. 

However. B.A.R. should be cautioned that the power of this compvter-based 
miAtlmedia simulation examination rests in the degree to which it accuratelu 
ff f" ?M l(?lf ffy ^ork on an actual vehicle and workplace envir onment. 

2. B.A.R. should include eztensiTC reliabiUtj analysis, through Kuder- 
Richardson formulas, Rasch analysis and calibration or other widely 
recognised and appropriate statistical method, in any future simulation test 
development, in order to insure the quality of the many elements of individual 
test questions. 

This study also contains important design recommendationa. 



DISCUSSION AND RECOMMENDATIONS 



INTRODUCTION AND BACKGROUND 



In May of 1986, the California State University system released a Memo of 
Understanding (#4), part of an Interagency Agreement with the California 
Department of Consumer Affaiis, Bureau of Automotive Repair (#12IA0077- 
B5), requesting 

a study be conducted on the fcMlbility of training smog check 
mechanics vim. hands-on ▼erification of mechanics ability to inspect and 
repair vehicles. The study should include recommendations on the most 
cost effectinre approach for conducting hands-on training, retraining and 
testing. 

The report produced from that contract, Hands-On Verification Of Mechanics 
Training: A Cost-Effectiveness Study Of Videodisc Simulation (Maher, 1988), 
recommended to the Bureau of Automotive Repair (B JV..R.) that: 

1. B.A.R. should no longer actually conduct mechanic training, but 
should continue to test, certify and license.... the Bureau should 
concentrate its resources on setting quality and performance standards 
for both the public and private institutions whose business it is to train 
and educate. These standards should include task lists of the 
knowledge, skills and abilities required by qualified mechanics to test, 
diagnose and repair emissions systems and the standards for instructors 
delivering training at public and private institutions. 

2. B.A.R. should test, certify and license mechanics through 
comprehensive "hands-on" simulation testing. While removing itself 
from the delivery of training, the Bureau should maintain control over 
the quality of mechanics through the use of the testing, certification and 
licensing function. ...Therefore, this study recommends Model 2, 
"Simulation Testing", as being the most effective method of insuring a 
hands-on, quality control capability for B.A.R., provided in the most 
cost-efficient manner.... [Simulation Testing] offers the advantages of 
standards setting, comprehensive testing and some insurance that 
mechanics can perform in a realistic environment. Because of the 
economies of delivery, this is the most cost-effective model ...comparing 
the total costs of all the models and the total costs per rtudent....the 
Bureau should [also] seek expert advice at the time of their actual 
solicitation. That advice would consist of the writing of very detailed 
specifications of the work to be performed for B.A.R., evaluating the 
submitted proposals, and reviewing work-in-progress. 

These recommendations were adopted by B.A.R., and began to be 
implemented by the development of a new, comprehensive Smog Check 
Certification Examination by B.A.R. in conjunction with the California 
Department of Consumer Affairs' Central Testing Unit. In addition, the 
California State Polytechnic University at Pomona (Cal Poly) received two 
subsequent contracts from B.A.R., parts of which were designated to 
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implement a "demonstration project" of computer-based, multimedia 
simulation testing for smog check mechanics. Part of the contract "Smog 
Check Program for the 90's" (Contract #12SA8101-B8. T. Maher. Project 
Director, November, 1988) was designated for the planning of a prototype of 
this simulation. The primary work product of that effort was a treatment 
document for the design and evaluation of the prototype simulation test 
(Maher & Harty, 1992, Appendix A-1.). 

Under a portion of the contract "Smog Check 2000: The 1991-1994 Bureau 
of Automotive Repair-Cal Poly Production Agreement" (Contract #12IA9040- 
Cl, L. Harty, Project Director, August, 1991), Cal Poly was to produce and 
evaluate the simulation test. Using B.A.R. personnel and others as content 
experts, Cal Poly initiated a process of test item development that resulted in 
a set of test item scenarios. In addition, two flow charts were developed, one 
for the actual paths of the examinee within each item and one for the 
progress of the entire test (Maher & Harty, 1992, Appendix A-2). In the fall 
of 1992, Cal Poly contracted with StatMedia (Gary W. Birch, President, Yorba 
Unda, CA) to develop and program the actual test, from this basic document 
set. 

This document reports on the experimental evaluation trials of the prototype 
simulation test, as compared to a "hands-on" test using actual vehicles. It is 
divided into major sections including this Introduction and Background . 
General Description of the Simulation Test. Evaluation Design. Results and 
Analysis. Discussion and Recommendations. References and Appendices. 

GENERAL DESCRIPTION OP THE SDIULATION TEST 

Simulation Test Components 

Software and Hardware Components 

The simulation test operates on an Apple Macintosh Ilci. with a single touch 
screen color monitor, controlling a videodisc player. TTie primary software 
controlling the simulation is MacroMedia Director ver 3.1. An entire list of all 
the individual software and hardware components of the simulation test is in 
Appendix B-1. 

Simulation Test Sections 

Introduction, Tutorial and Help: ITiis section orients the mechanic to the 
operation of the display device by introducing the computer and explaining 
the methods of both presentation and appropriate mechanic response. The 
"Tutorial" is limited to a description and demonstration of the interface 
components such as buttons and the "tools". In addition, this section shows 
very simple example questions and requires the mechanic to respond 
properly, whether by pushing a touch screen button or using a simulated tool. 
This "Tutorial" section is then used as the response to the "Help" button 
throughout the test section. 



Data Section: The data section of the test is unavailable to the mechanic. It 
contains the mechanics* logon information, including B.A.R. certificate 
number, the touch screen/keyboard entry responses of the mechanic to test 
questions, the completion time of each major item, the elapsed time for the 
entire tutorial and test, and the procedural sequence for question #23. At 
logoff, this data is exported by the MacroMedia Director program to a 
Microsoft EXCEL file. 

Test Section: The essential framework of this simulation examination is a 
smog check, including a visual inspection, functional check, and diagnosis of 
emission problems. Three vehicles were used: a 1989 Toyota Tercel, a 1990 
Buick Riatta, and a 1992 Ford Taurus. The actual test consists of nine test 
"items'* divided into visual inspection, functional inspection and diagnostic 
groups, with a total of 44 individual questions. The 44 questions were worth 
a total of 50 points. The items provide a general framework and beginning 
and end point for the mechanic. For example, "Item" #1, with 17 specific 
questions, provides the mechanic with a framework or context of the initial 
contact witii a vehicle. In addition, all of these questions require the 
mechanic to "look-up** information located on the vehicle itself, and /or in 
standard manuals provided at the test site. Appendix B-2, Test Item 
Summary**, provides a quick reference to the general content of the test 
items. Appendix B-3, "Smog Check Technician Checklist**, is a complete 
listing of the items and the full text of individual questions, with correct 
answers. In each area the examination includes "look-up" questions that 
require the use of actual manuals that were available at the test site. The list 
of manuals is contained in Appendix B-4. 

An on-screen "host" character acts as the guide through the exam, presenting 
questions verbally to the mechanic. The mechanic can also read the specific 
test question from the screen. Also, the host figure provides some transition 
between major elements of the test, and additional realism to the simulation 
environment. The host appears from both analog video from the videodisc, 
presented in a large window, or from digital video files stored on the 
computer hard disc, in a smaller window to the side. 

As appropriate, questions are fully simulated through audio /video 
presentations from the videodisc, and/or from digital audio/video/graphic 
files on the computer, shown on the single computer monitor. Mechanics are 
prompted for answers to questions on identification, selection of tools and 
methods, location of hookups for tools, and application of tools and methods, 
from the computer audio /video /graphic files, from a combination of 
computer-generated and videodisc sound and images, or directly from the 
videodisc source alone. Simulated tools that are available include vacuum 
gauge, timing light, oscilloscope with three optional display modes, and an 
exhaust gas analyzer. Also provided are simulated tachometer and 
temperature readings, engine starter, and accelerator pedal. In addition, the 
mechanic has the ability to move around the vehicle, and to "zoom in** to a 
number of areas Inside both the engine and the passenger compartments, to 
obtain closer views. 
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EVALUATION DESIGN 



General Design Considerations 

Hie general evaluation design was developed to examine the effectiveness of a 
computer-based, multimedia simulation test on California smog check 
mechanics. Both common sense and the research literature showed that a 
computer-delivered simulation test would be superior to a written 
examination. Consequently, comparing the simulation test to the current 
paper-and-pencil testing mode was not seriously considered. While not 
practical, the ideal testing methodology, given unlimited time and resources, 
would be to examine every mechanic with a "hands-on" test on an actual 
series of vehicles. Since the purpose of the simulation test was to approach 
this ideal testing environment, yet maintain a sense of practicality and cost- 
eflftciency, the simulation test was compared to a "hands-on" test. 

In addition, it was self-evident that the test population had to be current 
smog check mechanics, in order for the results of the comparison to have any 
validity with that group. Since both the prototype simulation test and the 
"hands-on" comparison test were outside the usual channels of B.A.R. 
examinations, currently certified smog check mechanics were to be recruited 
and paid to participate in the study. While this would be a self-selected, non- 
random sample of the population of certified smog check mechanics, the 
sample would be randomized between the two testing methods. The purpose 
of the cA^uation was the comparison of those two testing methodologies, not 
validating either test for use with the entire population. 

It should also be noted that most quasi-experimental designs with two groups 
are usually interested in comparing a "control group" with an "experimental 
group", where a "treatment" tiiiat is the object of Interest has been added to 
the conditions of the experimental group. In general, the question to be 
answered is whether there is a statistically signihcant difference in 
performance between the groups, a difference that might then be attributed 
to the treatment. However, in the current case both groups are considered 
the same. The question of interest is whether there is no significant 
difference in performance between the groups, suggesting that the two 
methods of examination are essentially equ^, and yield equal results. 

Finally, the evaluation gathered two kinds of data. In addition to the 
performance of the mechanics on the actual test questions, each group 
completed an identical, Likert-scaled questionnaire that asked them about 
their subjective response to the particular test methodology they used 
("Survey for Mechanics-BAR Demonstration Project", Appendix C). 
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Description of the "Hands-On" Actual Vehicle Test 



The creation of the "hands-on" actual vehicle test was based on one essential 
criterion: that it and the simulation test be as equivalent as possible. The 
major exception to this approach was that the "hands-on" test would not 
artificially restrict a mechanic's normal operating alternatives. Clearly, the 
simulation test did not provide the mechanic with all possible views of the 
vehicle, or every potential tool, or any test procedure desired, etc. The 
simulation did have a much more restricted set of options for the mechanic 
than the actual vehicle. However, for the "hands-on" test to reflect an actual 
work environment, it was determined that the mechanic could not be 
restricted, even If this became a source of differential performance between 
the test methods. 

The "hands-on" test did not contain a formal "Tutorial" or "Help" section. 
The test was continuously observed by the B.A.R. proctor, who would only 
answer procedural questions about the test process, but not about the content 
of questions. 

The test itself was essentially a "printout" of the nine test items and specific 
questions from the simulation test described earlier ("Smog Check 
Technician Checklist", Appendix B-3). The content, wording and order of 
the questions were the same. (Anomalies that appeared in the questions 
during the test administration are discussed under Data Collection below.) 
The exact three vehicles that were used to create the visuals for the 
simulation test were used for the "hands-on" test. With the exceptions noted 
in Data Collection below, the vehicles presented the same configurations, and 
required the same answers, as they did in the simulation test. In addition, all 
the tools that were available in simulated form on the "simulation" test were 
available and displayed for the mechanic, including a B.A.R 90 TAS analyzer. 
In this "hands-on" test environment the mechanic also had available an entire 
set of standard shop hand tools in a tool chest approximately five feet high. 
The same set of manuals available for the simulation was also available. The 
actual vehicle components photographed for the identification questions in 
the simulation test were displayed for identification questions in the "hands- 
on" test. 

The performance data were collected by having the mechanic fill out the 
blanks on the list of test questions ("Smog Check Technician Checklist", 
Appendix B-3). The checklist attached in the appendix also includes the 
correct answers. In addition, upon completion of the test each mech£inic 
completed the "Survey for Mechanics", (Appendix C). 

When the mechanics arrived at the test site, they were given the same 
introduction to the test situation that was part of the opening narration on the 
simulation test. They were given the test form to complete, shown the tools, 
manuals and displayed items for identification, and told to begin. The test 
had no time limit. 
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Sample Selection and Randomization 



The central issue in this study was the question of the equivalency of the 
simulation and "hands-on" tests. Consequently, the primary focus of sample 
selection was on randomizing the two test groups, rather than attempting to 
select test groups that would mirror the population of Smog Check mechanics 
throughout California. In addition, practical constraints of time and money 
made the use of a statewide sample impossible. For the purposes of this 
study, the basic assumption was made that there are no significant differences 
in the general abilities of Smog Check mechanics based strictly on their 
geographic location in California. Consequently, B;A.R. derived a random 
sample of 1,500 Smog Check technicians from a total of approximately 8,500 
in Los Angeles, Orange, Riverside, San Bernardino and Ventura Counties. 
(Appendix D-1, Memo Adelsperger to Harty, describes this process in detail.) 

B.A.R. then mailed letters (Appendix D-2, sample letter) to each of these 
1,500 technicians, asking them to indicate interest in participating in the 
BAR Smog Check Project by calling the Instructional Technology Ofiftce at the 
California State Polytechnic University at Pomona (Cal Poly). ^Approximately 
250 technicians called in to express Interest in participating in the project. 
At that time their current addresses were confirmed. Letters of response, 
with the form "Confinnation of Intention to Participate," (Appendix D-3, 
sample letter/form) and a self-addressed stamped envelope, were mailed to 
responding mechanics, requesting that they fill out and return the form by 
December 10, 1992. 

The 180 technicians who returned the confirmation form became the basic 
sample. This group was divided in half, based on the last digit of their Social 
Security numbers: Odd numbers were assigned to the "hands-on" test: even 
numbers were assigned to the simulation group. The first forty technicians 
from the unordered list of each group were scheduled. "Hands-on" test 
subjects were scheduled at four per day beginning on January 25, 1993, 
ending February 8, 1993: simulation test subjects were scheduled at six per 
day, beginning February 5, 1993 and ending February 15, 1993. 

At this time, four additional names were chosen as Beta test subjects for the 
simulation test. 

Letters with enclosed maps were sent to selected technicians, giving them 
time and location assignments (Appendix D-4, sample letter). Letters of 
regret were sent to those not chosen to participate in the project (Appendix 
D-5, sample letter). 

Trials were conducted at Citrus Community College (hands-on test) and Cal 
Poly Pomona (simulation test). Thirty-eight subjects were actually tested in 
each group. 
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Data CoUection and Presentation-Tests 

As described earlier, the raw performance data from the simulation test were 
accumtilated during the testing process and exported to a Microsoft EXCEL 
file. The record for each individual contained the unprocessed keystroke 
responses to each question. Some of those responses could have multiple 
wonl and number strings, making automated scoring by columns impossible. 
Consequently each record was manually scored and transferred to an answer 
grid (Appendix E-1). On that answer grid the raw scores from the nine test 
items are translated into 44 specific question responses. All answers were 
scored one point for a correct response, except for question #21 which was 
scored either two points for a correct answer, or zero points for an incorrect 
answer. Question #35 was worth six points, and scores could range from zero 
to six. Each of the individual checklists from the '*hands-on" test was also 
manually scored, and transferred to an suiswer grid in the same way 
(Appendix E-2). 

This manual scoring was also required of both tests because of specific 
problems that emerged with some questions. The initial question under item 
#5 of the "Smog Check Technician Checklist" (Appendix B-3), never actually 
appeared on the simulation test. Consequently the response was eliminated 
fi*om the "hands-on" test group during the hand scoring phase, and is noted 
as an "X" In the checklist. Because of problems of interpretation, on both the 
simulation and the actual vehicle, of whether a part is "missing" or 
"disconnected", and of contradictory information in commercial automotive 
manuals, questions #18 and #22 could have either of two correct answers. 

Because of technical problems in creating the actual defects in the "hands- 
on" vehicle for question #35, the malfunctioning cylinders were different in 
the simulation (cylinders 4&5) than in the "hands-on" test (cylinders 2&6). 
Because questions #36-#41 depend on this cylinder data, the answer order of 
the "hands-on" test was shifted to match the simulation. The nimiber of 
problem cylinders and the nature of the problems remained constant in both 
tests. In scoring, the answer order was conformed so that now question #39 
and #40 refer to malfunctioning cylinders on both the simulation and the 
"hands-on" answer grids. 

Because of an undetected wording difference between the simulation test and 
the "hands-on" test in question #43, both groups were scored correct if the 
mechanic was able to download any of the appropriate trouble codes. Because 
question #44 was dependent on that answer, it was scored correct for both 
groups if the mechanic was able to correctly find in the reference materials 
tiie meaning of a downloaued code. 

Data Collection and Presentatlon-Survev 

The data from the surveys for the simulation testing group (Appendix E-3) 
and for the "hands-on" testing group (Appendix E-4) were manually 
transferred to answer grids similar to those used for the test data. In addition 
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to the responses to the ten Llkert-scaled survey items, each mechanic was 
asked for their date of birth, their years of experience as a mechanic and as a 
certified smog check mechanic. Those data were also transferred to the 
answer grids. 

RESULTS AND ANALYSIS 

Test Reliability Results 

To properly evaluate the performance of the two groups on the simulation 
examination and the "hands-on"* version, the test itself had to be analyzed for 
reliability. If the test was not reliable, and contained excessive misfitting and 
unreliable items, then the performance comparison would not be very 
reliable. 

A Rasch analysis (Wright & Linacre, 1993) was chosen for the reliability study. 
This technique is currently used to norm examinations by the National Board 
of Medical Examiners and the American Society of Clinical Pathologists, 
among others. The basic specification of the Rasch analysis is that The more 
able the person, the more likely a success on any item. The more difficult the 
item, the less likely a success for any person" (Wright & Linacre, 1989). In 
addition to its currency, speed and simplicity, the Rasch analysis was also 
chosen because it could accommodate weighted items, such as questions #21 
and #35. The more traditional Kuder-Richardson formulas for test reliability 
require dichotomous scoring (right or wrong) (Borg & Gall, 1983). 

The process began by analjrzing the items for the simulation test and the 
"hands-on" test independently. No items fell outside the range of + two 
standard errors of measure. Consequently the groups were combined for 
additional analysis. The analysis also confirmed the visual inspection of the 
data that the test was relatively "easy" for both groups, because several items 
were correctly answered by all. 

The first analysis of the combined data found two items that significantly 
"misfit": questions #23: "Results of the functional EGR system test"; and 
#26, "Identifying component C, the EGR position sensor". The items 
"misfitting" had inconsistent responses that were outside the range of 
responses expected of items of that difficulty. Also, three persons were found 
to significantly "misfit". Persons "misfitting" responded inconsistently 
outside the range of responses expected for persons of their ability. These 
items and individual were subsequently removed from the analysis. In 
addition, items that were answered correctly by all, and individuals who 
answered all items correctly, were eliminated. 
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The test was then recalibrated. The original results of the first analysis of the 
combined group, and the final combined recalibration are shown below: 



Original Calibration: 


Person Separation Reliability 


=.52 


Item Separation (Test) Reliability 


=.82 


Final Calibration: 


Person Separation Reliability 


=.60 




Item Separation (Test) Reliability 


=.86 



The person separation reliability illustrates the earlier comment that the test 
was relatively easy. This would, however, be expected in this type of mastery 
testing situation with a self-selected sample. In addition, the test was 
designed specifically to illustrate various types of item possibilities, and not to 
discriminate and distribute abilities among this group. 

The high item separation reliability, or test reliability, indicated that in 
general persons responded consistently to the test according to their 
performance capabilities. 

Test Group Comparison Results 

Once the test reliability was determined, all items and all scores were used in 
the comparison of the performance between the simulation test group and 
the "hands-on" test group. The summary statistics for the raw scores are 
shown below: 



RAW SCORES 


Number Mean 


Standard Deviation 


Simulation Test 
"Hands-On" Test 


38 44.184 
38 43.737 


3.525 
3.117 


t = .586 


Degrees of Freedom = 74 


Probability = .560 


The Rasch analysis provides a calibrated measure equivalent for each raw 
score. The summary statistics for the measures are shown below: 


MEASURES 


Number Mean 


Standard Deviation 


Simulation Test 
"Hands-On" Test 


38 2.766 
38 2.616 


1.253 
1.238 


t = .525 


Degrees of Freedom = 74 


Probability = .601 



Based on the relatively low "person separation" reliability noted above, there 
was some concern about the possibility that the test contains too many "easy" 
items, and that the "inflated" scores by both groups would mask any 
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dififerences in the presentation and delivery methodology. As a check against 
this possibility, any question on which there was a combined score between 
groups of 76 (all correct: 38 & 38) or 75 (1 incorrect in either group) was 
eliminated from the total individual scores. A total of 14 questions were 
eliminated: #1, 4, 5, 6, 7, 8, 11, 14, 15, 16, 19, 22, 27, and 28. The concern 
about the degree of diifBculty of the test was unfounded, as the scores of the 
two groups actually became closer. 



RAW SCORES minus "Easy" Items 

Number Mean Standard Deviation 

Simulation Test 38 30.211 3.378 

"Hands-On" Test 38 29.974 2.936 

t - .326 Degrees of Freedom = 74 Probability = .745 
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Clearly, the differences between the two groups are not statistically 
significant. Hiese data all show that the probability is in fact very high that 
the differences between the group means are the result of chance. 

The "t test" is a standard formula for statistical analysis to compare the 
differences between two groups, and can be found in almost any inferential 
statistics textbook. These textbooks usually have tables of the basic t values. 
The tables also indicate various "degrees of freedom" and various probability 
levels. Ihe degrees of freedom for a statistical test relates to the nimiber of 
persons or scores used in the test computations. In general, fewer 
persons/scores require higher computed t values for any dififerences to be 
significant. In the actual analysis, the t value is computed from the scores of 
the groups. The degrees of freedom in the two-sample case are the number 
of persons in each group (38) added together (76) minus two, one for each 
sample (74). Also, it should be noted here as a reference point on the 
probability figures that if the research question was focused on finding a 
difference between groups, in order, for example, to support the worth of a 
particular treatment received by one of the groups, then the probability level 
would typically be set at p <.05, or even p <.01. That is, in advance of the 
experiment, the researcher would require that the probability (p ) be less 
than (<) .05, or even .01, that whatever dififerences may be found are the 
results of chance influences. In these cases t values that would yield larger 
probabilities, such as .06 in the first case or .02 in the second, would be 
considered non-significant dififerences because of the potential influence of 
chance. Consequently, t values with probabilities of the magnitude in this 
study ( p S.560, p s.60l, p s.745) indicate imambiguously that the dififerences 
are non-signifiicant. 

Survey Results 

The results of the opinion survey for both groups are listed below as the 
average response value: 
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1» strongly agree 2« agree 3s undecided 4« disagree 5b strongly disagree 

Simulation Hands-On 

a This test gave me a fair chance to show my 
skills as a "Smog Check" mechanic. 



b. This test was very easy for me. 

c. Most of the questions on this test were very much 
like the things that I really do on my Job. 

d. This test gave me a fair chance to show my 
knowledge of the "Smog Check" program. 



1.95 
2.50 

1.84 

1.87 



e. Most mechanics that are certified for the "Smog 

Check" program would say that this was a hard test. 2.79 

f. I think that all the "Smog Check" testing should be 

done Just like the test that I Just took. 2.4S 

g. The best kind of "Smog Check" test is the method 

using a written test, pencil and answer sheets. 3.55 

h. The way that the test I Just took was given to me 

was very confusing. 3.55 

i. I enjoyed taking a test using this method. 1.76 

J. Most of the things that this test asked me about I 

don't usually do on my Job. 3.89 



1.66 
2.24 

1.87 

1.68 

2.95 

1.92 

2.SZ 

4.03 

1.63 

3.55 



In addition, the descriptive information about the mechanics is summarized 
below: 



Age (as of Dec. 31, 1992) 
Teais as a mechanic: 

Tears as a certified smog check mechanic: 



38.2 
14.1 
8.21 



37.3 
13.5 
8.41 



The summary statistics for the survey averages are shown below: 



Number Mean 



Simulation Test 
"Hands-On" Test 

t = 1.056 



10 
10 



2.258 
2.050 



Standard Deviation 

.443 
.437 



Degrees of Freedom =18 Probability = .305 
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In order to compUe these summary statistics, the actual values for the 
"negatively" stated items, #e, #g, #h, and #J, were reversed for the 
computations only. Otherwise the higher values for these "negative" items 
would have masked positive values. 

ITiese statistics show that the differences between the two groups is not 
significant. However, an initial Rasch analysis of the individual survey 
response items indicated significant differences between the groups on items 
#f, #g, and #h. Item #j, which was the direct reverse of item #c, was 
apparently confusing and was also a misfit in the analysis. 

DISCUSSION AND RECOMBSENDATIONS 

Discussion of Test Results 

The purpose of the B.A.R. Demonstration Project was to examine the 
feasibility of creating a computer-based multimedia simulation examination 
that could provide B.A.R with the same level of confidence in assessing the 
ability level of mechanics as a "hands-on" actual vehicle examination, but in a 
more cost-eff'ective manner. Hie statistical comparison of the certified smog 
check mechanics that were examined with the prototype simulation test, as 
compared with an equivalent "hands-on" test, indicate that the simulation 
test is not significantly difierent from the "hands-on" test. These findings are 
congruent with, and supported by, the general literature on simulation 
training and testing (Maher, 1988): 

In summary, it seems clear that computer-controlled videodiscs can 
be an effective method for delivering training and creating simulations. 
...Videodisc training has been shown to work effectively within the 
automotive industry in the delivery of training to auto mechanics. 
Finally, state agencies have used videodiscs in both their training and 
Hcensing functions.... Videodiscs are the least expensive delivery 
method because of the elimination of instructors and the reduced 
training time required to complete coursework. These conclusions are 
confirmed by the reviews of literature that discuss cost-effectiveness 
issues (Azia, 1986; Bosco, 1986; Russ-Eft, 1985; Orlansky & String, 
1979, 1981; Spuck, 1981; Van der Drift, 1981; Wilkinson, 1980). 
Videodiscs therefore provide the most cost-effective delivery method 
for hands-on mechanic training and vaiflcadon testing, [italics added] 

In addition, there were no data in the survey results discussed below to 
indicate any extreme opinions expressed by the mechanics concerning 
resistance to the simulation examination method. As noted in the BA.R cost- 
effectiveness study (Maher, 1988) that originally discussed the advantages of 
simulation testing, 

B.A.R. should continue in its quality control function by creating a 
comprehensive examination procedure to verify the attainment of 
those standards of performance. That examination should be a 
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videodisc-based, vehicle simiilation test that will enable the Bureau to 
verify mechanics' ability to operate in the vehicle environment. 

Consequently, since the performance data between the simulation and 
"hands-on" methods of testing have now been shown to be not significantly 
different to a hij^ level of confidence, this evaluator recommends that: 

1. BJLR. should convert its cuxrent written enmination to a compiiteHbased 
multimedia simulation examination, using the cuirent demonstration project 
design documents and wrtsHng simulation test, with at least the modifications 
recommended below, as the working proto^pe. 

However, B^.R. should be cautioned that the power of this cowputer-based 
mulWnedia simulation examination rests in the degree to which it accurately 
stnvulates the work on an actual vehicle and workplace environment. While 
obviously more convenient, and with other administrative advantages, the use 
of the computer only to deliver what is essentially a "text" examination, with 
a few visual embellishments, is to continue to use a "written" test, without 
the strengths of multimedia simulation. 

The analysis of the test items found two questions that were significantly 
"misfitting" in relationship to the rest of the questions, and several of the 
questions that did not discriminate, this is, everyone answered them 
correctly. Because of the potential for difficulty in understanding parts of the 
computer simulation inteiface, as well as to document that only appropriate 
and useful questions become part of any fiituure simulation test item bank, this 
evaluator recommends that: 

2. B.A.R. should include extensive reliability analysis, through Kuder- 
Richardson formulas, Rasch analysis and calibration or other widely 
recognized and appropriate statistical method, in any future simulation test 
development, in order to insure the quality of the many elements of individual 
test questions. 

Discussion of Survey Results 

While no statistical tests were used, observation would indicate that the data 
on the age, average years of experience as a mechanic and as a certified smog 
check mechanic, show the two groups to be essentially equal on those 
measures, confirming the success of randomizing the original sample. 

The survey results indicated that there was no significant difference between 
the opinions of the "Simulation and "hands-on" groups concerning their 
respective testing eiqperiences. In this case the opinions of the "hands-on" 
testing group acted as a "control" group for those of the simulation group. 
The importance of this finding is that the general subjective reaction of the 
simulation group to their experience does not seem to contain any issues that 
would inhibit the acceptance of simulation testing as a new test methodology 
for BAR 




Both groups tended to either strongly agree or agree with the statements that 
said: 

• This test gave me a fair chance to show my skills as a "Smog Check" 
mechanic; 

• Most of the questions on this test were very much like the things that I 
really do on my job; 

• This test gave me a fair chance to show my knowledge of the "Smog 
Check" program; and 

• I enjoyed taking a test using this method. 

Both groups tended to be undecided, on average, about statements that said: 

• Most mechanics that are certified for the "Smog Check" program would 
say that this was a hard test; or that 

• This test was very easy for me. 

In addition, both groups were apparently confused about the reversed, 
negative wording of the following statement and responded inconsistently: 

• Most of the things that this test asked me about I don't usually do on my 
job. 

However, the groups did diverge on three of the statements. First of all. the 
simulation group tended to be, on average, more undecided than the "hands- 
on" group about agreeing that: 

• I think that all the "Smog Check" testing should be done Just like the test 
that I just took. 

In addition, the simulation group tended to be, on average, more undecided 
than the "hands-on" group about disagreeing with the statements that: 

• The best kind of "Smog Check" test is the method using a written test, 
pencil and answer sheets; and that 

• The way that the test I just took was given to me was very confusing. 

Taken together, the reaction of the simulation group to these three 
statements would seem to indicate some reduced acceptance of, and some 
increased confusion about, the simulation testing method compared to the 
reactions of a similar group of mechanics to the "hands-on" method. Given 
the general and long-term famiUarity of smog check mechanics with the 
"hands-on" testing environment of working on actual vehicles with resd tools, 
and the "newness" and /or "strangeness" of the computer simulated 
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environment, it is somewhat surprising that the magnitudes of these 
differences of opinion between the groups are actually not much greater. 
However, because the functioning of the simulated vehicle environment, the 
methods of working with simulated tools, and the other necessary constraints 
of the simulation test, have the clear potential to increase the test resistance 
and hei^ten the confusion of some mechanics, however small that increase 
might be, this evaluator recommends that: 

3. B.AJL should insure that the follj configured simulation test incorporate a 
complete^ developed tutorial and help section, with at least the features 
described in the original design document (Appendix A-1). 

A completely developed tutorial and help section would also assist B JLR. by 
increasing the probability that the fully configured simulation test would be 
able to ftmction as a ''stand-alone** examination workstation, without the need 
for full-time test proctors. Additional support for this recommendation is 
discussed In the first two items below under Evaluator*s Observations . 

It should be noted that the simulaOon test proctors, Sam Hay ami Larry Harty, 
who were also the prime test developers from The CaUfomia State 
Polytechnic University at Pomona^ have indicated that they feel that the 
1\itoriar section can be signtficantly reduced (Appendix F-1, item #7). 

Evaluator*s Observations and Recommendations on Simulation Test Design 

(On February 4-5, 1993 the evaluator visited the sites of both the "hands-on- 
test and the simulation test. He observed one ''hands-on" examination and all 
or most of six simulation examinations. In addition, the evaluator went 
through the entire simulation examination process as an examinee. 
Observations by the Cal Poly test proctors, and the "hands-on" proctor, Phil 
DeLeon, a BJV.R. employee and test developer, are attached in Appendices F-1 
and F-2 respectively. These documents also present a number of important, 
additional design recommendattons ) 

A When a mechanic was unable to complete an item, for whatever reason, 
yet continued to make attempts, the proctors for both the "hands-on" and 
simulation tests would eventually interrupt the process, fail the mechanic on 
that question, and have the mechanic proceed with the test. These were 
generally subjective Judgments made by the proctors. This behavior was 
observed in both test groups. While the action of the proctors could have 
theoretically had an influence on the eventual outcomes of the tests, 
discussions with the proctors led to the conclusion that any possible effects 
canceled between groups. An expanded "Tutorial" section should provide 
instructions to the examinee concerning appropriate behavior in these kinds 
of situations. 
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E The simulation test proctors were occasionally required to coach 
mechanics throu^ the "Help" module itself, as well as to provide assistance 
to the mechanics with the operation of the computer interface during the 
actual examination. Difficulties were observed in both the use of the buttons, 
and with the "point-and-drag" simiilated tools. Part of the interface problem 
is the result of the relatively slow response speed of the simulation test when 
compared to the quicker responses on the BJV.R. 90 TAS analyzers with 
which these mechanics are all familiar. Speeding up the responsiveness of 
the simulation would be very helpful. However, many of the interface 
problems were the result of an underdeveloped Tutorial" section at the 
opening of the test. As noted in the original design document, examinees 
should not be able to proceed to the actual test untH they have successfully 
demonstrated mastery of the interface through completion of example 
questions in the "Tutorial" section. 

Both this and the previous item provide additional support to the 
recommendation made above concerning the expansion of the "Tutorial'' 
section. 

C Both tests contained an "experimental" test question that required the 
mechanics to complete a work order form based on information developed to 
answer the questions numbered #35-#41. It became clear very early in the 
administration of the examination that not only was the computer 
programming not refined enough at that point to score this text question on 
the simulation test, but that the wording of the question itself made scoring 
problematic even on the "hands-on" test. This question was disregarded 
before formal scoring began. However, the results were such that a different 
kind of question that required a more precise "flll-in-the-blank" text 
response could be computer scored. In addition, another computing 
experiment was attempted in which the nimiber of steps and the sequence of 
steps were recorded by the computer for a mechanic's progress through the 
EGR procedure required in question #23 of the simulation test. Also, the 
computer recorded the start and stop times for each item. Since Hiere was 
no comparable records for the "hands-on" test, these data were not used, but 
indicate some of the wide range of additional information that ipan be 
collected through computer delivery of tests. Therefore this evaluator 
recommends that: 

4. B.AJL should insure that the faXtj configured simulation test incorporate as 
much data collection, and as wide a varietj of simulation test questions* as 
they think may be useful, without prejudging the actual feasibiUty of either. 

D. With the exceptions noted earlier for the "experimental" questions, the 
"Data Collection" section of the simulation test provided a file that contained 
only raw data. Consequently, each person had to be manually scored. This is 
obviously a costly and time-consuming process. While it is critically important 
for the raw score data to be available for quality control in the scoring 
process, this evaluator recommends that: 
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5. B JLR. should insiue that the faOy eonBgaxtd simulatioii test Incoipoiate a 
"Data" section that at least scores correct answers to questions, and also may 
provide mious reports on both questions and examinees, such as subscores 
on Tisual inspection, functional inspection, diagnosis, etc.. all available via 
remote tdecmnmunications access. 

6. Biinor Design Recommendations: Hie following are some less critical 
observations on the simulation test, and corrective suggestions. 

Unlike the "hands-on" test, where the answers to previous questions were 
always available, the simulation prevented looking at earlier answers. That 
restriction caused some unnecessary delays because that earlier information 
was needed to answer later questions. 

• The fully configured simulation test needs a continuing "title bar" or 
other on-screen prompt on data already gathered and "known." such as 
vehicle model, engine size etc. This could even be a "pop-up" electronic 
"scratch pad" on vrbleh the mechanic could write notes. 

Because of understandable restrictions in the prototype simulation test, 
questions do not remain on the screen when any of the "tools" are used. 
There were occasions when the mechanic was unsure of the exact question 
wording. Also, the "Help" button is often replaced with other information. 

• The fully configured simulation test needs to have the question present 
on screen at all times until it is answered, and the "Help" button present on 
screen at all times. 

Finally there is the broad design issue of how much to "focus" or "restrict" 
the choices of the examinee, relative to an actual vehicle environment. In the 
"real" environment, mechanics may have a large number of possibilities for 
arriving at a "correct" answer to a problem. While it mi^t initially appear 
that the goal of the test is correct answers, B.A.R would seem to want those 
"correct" answers found by the use of a particular, specified "correct" 
procedure. For example, on an actual vehicle some mechanics could "cheat" 
the EGR test of vacuum by pressing with their fingers on the EGR diaphragm, 
instead of using a vacuum gauge. Another example is the use of a paper clip to 
pull basic trouble codes. However, this method misses much of the electronic 
diagnostics available through factory methods of accessing trouble codes. The 
point is that whUe not exactiy the same as the actual vehicle work 
environment, the simulation test could exclude these possibilities. 

• The fiil^ configured simulation test should provide B.A.R. the opportunity 
to enforce its interest in the use of recommended and appropriate smog 
check and diagnostic procedures, even at the expense of causing the 
simulation work environment to differ from the actual work environment. 
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Summary of Recommendations 



General Recommendations: 

1. B^R. should convert its current written examination to a computer^Msed 
multimedia simulation examination, using the current demonstration design 
documents and simulation test, with at least the modifications discussed 
below, as the working proto^e. 

2. B^R. should include extensive reliability analysis, through Kuder- 
Richardson formulas, Rasch analysis and calibration or other widely 
recognized and appropriate statistical method, in any future simulation test 
development, in order to insure the quality of the many elements of individual 
test questions. 

Design Recommendations 

3. B^R. should insure that the faOy configured simulation test incorporate a 
completely developed tutorial and help section, with at least the features 
described in the original design document (Appendix A-1). 

4. B^R. should insure that the fuUy configured simulation test incorporate as 
much data collection, and as wide a variety of simulation test questions, as 
they think may be useful, without prejudging the actual feasibility of either. 

5. B.AJEL should insure that the fully configured simulation test incorporate a 
"Data" section that at least scores correct answers to questions, and also may 
provide various reports on both questions and examinees, such as subscores 
on visual inspection, functional inspection, diagnosis, etc., all available via 
remote telecommunications access. 

6. BOnor Design Recommendations: 

• The fiilly configured simulation test needs a continuing "title bar" or 
other on-screen prompt on data already gathered and "known," such as 
vehicle model, engine sise etc. This could even be a "pop-up" electronic 
"scratch pad" on ^ddch the mechanic could write notes. 

• The fully configured simulation test needs to have the question present 
on screen at all times untU it is answered, and the "Help" button present on 
screen at all times. 

• The fully configured simulation test should provide B.A.R. the opportunity 
to enforce its interest in the use of recommended and appropriate smog 
check and diagnostic procedures, even at the expense of causing the 
simulation work environment to differ from tiie actual work environment. 
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SMOG CHECK FOR THE 90*8 



DEMONSTRATJON PROJECT 

Treatment Statement for 
Simulation Testing 



I. ASSUlflPTIONS 

A. General 

1. This treatment is for a demonstration of the ability of computer- 
assisted, videodisc-based media to create simulations of the elements 
of a smog inspection to test smog check mechanics for certification. 
With these videodisc simulations, it will be possible to test and 
evaluate the ability of this particular delivery method to assess the 
knowledge, skills and abilities of smog check mecha^iics, as compared 
to assessing the same general level of performance of mechanics by 
using actual vehicles. If the test hypothesis is cbrrect, that the 
vide(Klisc simulation method is at least equivalent to the actual vehicle 
method of assessment, then BAR can use certification testing through 
videodisc simulation, without the time and expense of testing 
mechanics on actued vehicles. 

2. This testing methodology will allow mechanics to answer questions 
at their own pace in an individual setting. This will permit the 
mechanic to test in more than one session, and at a time convenient 
to the mechanics and their employers. 

3. This demonstration will be evaluated by comparing the 
performance of mechanics in the videodisc simulation test 
environment to the performance of mechanics in similar test 
situations on actual vehicles. Tliis general evaluation procedure will 
require that actual smog check mechanics be hired (approximately 
$80 per mechanic as recommended by BAR) to participate as 
evaluation subjects, and that the evaluation occur outside the normal 
paper testing cycle. 

B. Hardware Components 

The original working assumption of this treatment is that the final 
demonstration will operate on an Apple Macintosh II with color 
computer monitor, controlling a videodisc player with a separate, 
color video monitor. 

However, recent advances in the display of video material within the 
computing environment, i.e. on the same monitor as computer data, 
may be more cost effective. Nothing in this treatment is meant to 
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require a two-monitor display situation, nor require the use of 
videodisc over any other optical storage media such as CD-ROM, DVI, 
etc. It is hoped that only minor modifications, if any, would be needed 
in this treatment to accommodate these optional presentation 
situations. 

C. Software Components 

The working assumption of this treatment is that the final 
demonstration will be written completely in -/^ple's HyperCard 2.0 (or 
hi^er) software. 

However, because of the requirements of test scoring and data 
collection it may be necessary to use a secondary database program to 
perform these functions within the HyperCard shell. Such a database 
would of course be transparent to the mechanic user. 

n. MODULES IN THE DEMONSTRATION TEST PACKAGE 

A. General Requirements 

The customized software components that will be created for this 
demonstration project will function in support of the following set of 
tasks and knowledge requirements originally developed by BAR: 

BAR OBJECTIVES 

VeUcle I.D. 

1. Identify engine configuration and Smog Check 
requirements by examining underhood emissions 
label. 

2. Identify engine and required ECS by referring to 
VIN and manusds if emissions label is missing. 

3. Determine vehicle certification status (federal, 
California, Canadian, none) by referring to imderhood 
label, reference manuals and emission control 
equipment. 

4. Review emission label to determine ECS and 
emission devices required on the vehicle. 
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^lud Inspectioii 

1. Inspect vehicle for missing, modified, 
disconnected or defective (MMDD): 
a PCV system 

b. Thennostatic air cleaner 

c. Evaporative controls 

d. Catalyst 

e. 3-way catalyst 

f. Exhaust gas recirculation system 

g. Ignition spark controls 

h Feedback/computer control system 
i. Air injection S3rstems 

Functional Test 

1. Perform timing check to determine if engine 
timing is within specifications. 

2. Test EGR system for proper functioning by 
applying vacuum, blocking esdiaust, etc. 

3. If engine failure light is in operation, extract 
codes and enter into TAS. 

Dia^osis 

1. Interpret results of TAS printout to determine 
cause for test failure. 

2 Interpret wiring and vacuum diagrams. 

3. Determine cause for high HC emissions by 
checking electrical, vacuum, or mechanical systems. 

4. Determine cause for high CO emissions by 
checking fuel components. 

5. Perform mechanical diagnosis of engine by 
checking for leaking valves, worn piston rings, 
clank, smoke, oil drip, etc. 

6 Diagnose/repair the following systems: 

a. exhaust gas recirculation (EGR) 

b. computer control 

c. fuel injection 

d. ignition 

7. Perform closed loop confirmation test. 
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In addition, all the modules in this test package should have the 
following on-screen navigational devices to help users: 

1. An on-screen graphic device on the border of the computer frame 
and the matching video frame, orienting users to their place in the 
program at all times; 

2. Constant and consistent on-screen "command line" icons that will 
allow the user to go to "help", to escape /quit, to return, etc., from 
every frame in the program. The user should not feel "trapped" at any 
point in the program. 

B. Mechanics Use Only 

1. Logon/Logoff: This is a very short module (subroutine) that 
would present a screen for the mechanic requesting identification 
data, providing a start/stop time for that session on the test system, 
and routing the mechanics to the appropriate section of the test based 
on their previous logon history. 

2. Introduction: This module would orient the mechanic to the 
operation of the display device by introducing the Apple Macintosh 
equipment and explaining the methods of both presentation and 
appropriate mechanic response. In addition, this module will show 
very simple example questions and require the mechanic to respond 
with the proper device such as keyboard or mouse. The mechanic will 
not be able to enter the actual test module without having responded 
in a way that shows an understanding of the test and correct response 
method. This introductory module will be mandatory for first-time 
users, but can be bjrpassed at latter logons. 

3. Help: Based on the material from the introduction, as well as 
additional material that may be required as specific test items within 
the test scenario are developed, this module should always be available 
to the student throughout the test. To the degree that it is feasible, 
the help should be context-sensitive. However, the help should not 
provide any additional information about any specific question on the 
examination, but only on the method of the exam and appropriate 
response options. The help should not favor those who may be 
computer literate enough to use it to gain an advantage, but rather the 
help should be designed to remove any "interference" in the test 
process caused by the computer-assisted videodisc methodology. The 
goal is to put the mechanic in as direct a contact as possible with the 
content of the simulations. 

4. Test Scenario: This module will take the mechanic through 
much of a "standard" inspection scenario, and require the mechanic to 
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answer questions throughout. (This is described in detail in section 
III. below.) 

C. Bar Use On:^ 

1. Data module: This module will have several component parts, 
such as 

a. Individual identification and logon data. This material will track 
how many sessions were required to complete the test, how long each 
session was. what time of day the sessions occurred, how often and 
where the mechanic used "help", etc. 

b. Scoring. The test will be divided into sections that correspond to 
the current categories in the BAR certification examination, and the 
computer will track and report the mechanics' scores in each section 
at the end of the test. The report should be at a level of detail that 
includes the identification of the missed test item(s), and the category 
of each missed item. 

c. Data analysis. If feasible, this part of the module should be able to 
compute and report some elementary score analysis to BAR for a 
particular duration of time, for a particular nimiber of mechanics, etc. 

2. Security: This module would insure the overall integrity of the 
examination by such actions as preventing mechanics from returning 
to previous questions and to previous sections of the exam in 
subsequent logons, insuring the security of individual test scores as 
well as of the answer keys, and other related security issues. 

3. Communications module (Optional for demonstration): This 
module would provide the abfiity for each testing system that is 
eventually distributed around the state to communicate online with 
BAR. The primary use of this communications would be to send the 
information from the data module to BAR. However, it could be 
possible to use this long-distance communications channel to update 
the computer portions of the test by changing the screen graphics. 
Because of the potential for additional software and hardware expense 
that may not have been anticipated in the development and production 
budget, it should be noted that this communications module is not 
required for the demonstration. It mi^t become a component on the 
testing package only in a full, state-wide implementation. In the 
demonstration, the data could be collected manually from the hard 
disc 

m. DESCRIPTION OP THE TEST SCENARIO 

The essential framework of ithis simulation examination will be a smog 
check, including a visual inspection and functional check. In each 
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area the examination will include "look-up" questions that require the 
use of actual manuals that will be available at the test site. 

The smog check scenario would be anchored in a specific garage 
environment, using a specific make and model of automobile to 
perform the check. In addition, a "host" or automotive supervisory 
character would act as the guide through the exam, presenting 
questions verbally to the mechanic as well as having the mechanic 
read them from the screen. Also, the host figure will be able to 
provide transitions between major elements of the test, and provide 
additional realism to the simulation environment. 

As the mechanic proceeds through the smog check sequence, the 
"supervisor" will ask appropriate questions about the vehicle that has 
been brought in for the check. However, the supervisor can also ask 
additional questions about the same problem on other vehicles, which 
can be shown directly to the mechanic. For example, after asMng the 
mechanic to identify a tamper of the PCV system on the original 
scenario smog check vehicle, the supervisor could then ask about the 
same system on a different vehicle, which could be shown to the 
mechanic. The framework would be a simple hypothetical "What if 
this were a 1984 Toyota Celica...." 

The specific mcmi^acturer, model and year qf vehicles to be usedf as 
well as dettdled descriptions qf the prdblem conditions to be tested 
within this scenario framework, will be provided by BAR, 

Since the mechanic cannot return to earlier questions in the test, the 
supervisor is able to ask additional questions that might of necessity 
provide information about an earlier test item. 

The actual test questions would be fiilly simulated through audio/video 
presentations on the videodisc, shown on the television monitor. 
Prompts as appropriate for questions on identification, selection of 
tools and methods, location of hookups for tools, and application of 
tools and methods, could come from a combination of the computer 
screen or directly from labels in the video material. 

Finally, the use of the TAS could be integrated into the entire smog 
check sequence. Mechanics might also have the option of selecting 
the TAS analyzer that they work with. The combination of computer 
prompts, and sound and picture information on the videodisc, could 
eliminate the potential differences that might exist in screen displays 
and layouts of the various models of TAS analyzers that the mechanic 
could encounter in the actual work environment. 
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IV. EVALUATION 



This plan describes the evaluation of the capability of computer- 
assisted, videodisc-based media to create realistic simulations of the 
elements of a smog inspection, for the purpose of testing smog check 
mechanics for certification. The working hypothesis is that this 
particular delivery method (videodisc simulation) will permit as 
accurate an assessment of the knowledge, skills and abilities of smog 
check mechanics as having them examined by using actual vehicles 
with BAR examiners. If this hypothesis is correct, that the videodisc 
simulation method is equivalent to the actual vehicle method of 
assessment, then BAR can conduct certification testing through 
videodisc simulation without the time and expense of testing 
mechanics on actual vehicles. 

This evaluation will be done by comparing the performance of 
mechanics in the videodisc simulation test environment to the 
performance of mechanics in similar test situations on actual vehicles. 
This general evaluation procedure will require that actual smog check 
mechanics be hired (approximately $80 per mechanic, as 
recommended by BAB) to participate as evaluation subjects, and that 
the evaluation occur outside the normal paper testing cycle. 

For the purpose of this evaluation plan, it is assumed that the 
computer-assisted, videodisc simulation software will be able to 
accumulate and report the responses of each mechanic to each test 
item. 



A. General Methodology 

The effectiveness of the evaluation rests on the ability to compare the 
group doing the simulation testing with a similar population that was 
tested on the same content, but delivered through a different 
methodology. This is a straightforward, quasi-experimental design 
that will require an experimental group and a control group. The 
experimental group will be tested using the computer-assisted, 
videodisc simulations, the control group will be tested on actual 
vehicles, and the results of the groups will be statistically compared 
and analyzed. The experimental hypothesis is that there will be no 
difference between groups. 

1. Sample Size, Selection and Randomization 

a. The minimum recommended size for the sample is 80 smog check 
mechanics, divided into the two groups of 40 each. In this type of 
evaluation design, the statistically important minimum number is 
usually considered to be 30 per group. The additional ten individuals 
in each group provide a practical cushion for absentees or other 
participation problems. Larger groups would increase the ability of the 
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evaluation to detect differences in performance between groups, but at 
a significant increase in cost in both participant fees and test 
administration. Fewer than 30 participants per ^oup may cause the 
masking of performance differences between groups. 

b. The practical problems concerning the number of videodisc 
systems to deliver simulation, and the locations where it is feasible to 
set up an actual vehicle test site, limit the geographic area that can be 
used to select a sample. While this geographic limitation, and the 
necessity of recruiting a paid sample, may introduce some theoretical 
problems in generalizing from the results of this evaluation design to 
the entire population of smog check mechanics, these potential 
sources of error are minimal. On logical grounds it can be argued that 
the motivations of the self-selected mechanics who participate for the 
money will balance out across the population. In addition, it will be 
possible to examine the demographics of the sample population (i.e.; 
age, years of experience, type of employer, geographic location of 
employment) against the demographics of the entire population to 
note any large scale deviations in the make-up of the sample. Finally, 
the randomization of the sample into the experimental and control 
groups will provide the rationale to compare their performance in 
each situation, which is the point of the evaluation. 

c. It is recommended that the selection of the sample begin with a 
letter from BAR soliciting participation, noting the fee, esqilaining the 
time parameters of participation, and indicating that only the first 80 
that respond with be selected. Once the entire group of 80 has been 
identified, they must randomly be divided into experimental and 
control groups. This randomisation into groups is a critical phase for 
the validity of the evaluation. Consequently, the randomization must 
be done with strict adherence to a formal table of random numbers, or 
some other equally formal and reliable method. It should also be 
noted that the administrators of the evaluation should maintain 
contact with the selected mechanics with follow-up letters and phone 
C£dls, confirming their willingness to participate, the time(s) and 
location for the test, etc., in order to maintain the integrity of the 
sample size during the emluation process. 

2. Test Content 

a. Another critical issue in the evaluation process is the translation of 
the simulation test items to an actual vehicle. Because of the nature of 
the simulation technology, it may be a "temptation' to create 
interesting test items that work well in simulation, test an important 
knowledge, skill or ability, but which may be very awkward, 
cumbersome, time-consuming, difilcult and expensive to recreate in 
an actual vehicle environment. In order to eliminate problems in the 
evaluation, test items must be carefully analyzed in the development 
phase to insure their feasibility and practicality on a vehicle. In 
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addition, it is assumed that all the simulated test items will be part of 
the actual vehicle test, and will parallel the simulated items as closely 
as possible. This extends to the necessity of writing explicit and 
matching instructions that the vehicle test administrator will deliver 
to the subject, and training the vehicle test administrators in the 
degree of help, if any. that they are allowed to provide, and the 
necessity to strictly adhere to the uniform delivery of instructions to 
all subjects in the control group. This problem does not exist for the 
experimental group because Uiey will be uniformly tested by the 
computer-assisted simulation system itself. 

b. In addition to the results for the smog check test items, each group 
should be asked to complete a series of 10-15 Likert-scaled items that 
pose questions to the participants about their subjective impressions 
of the test. The primary purpose of this questionnaire is to gather 
data about the feelings of the subjects to their experience with the 
videodisc simulation delivery. However, for general purposes of 
comparison, and to provide a baseline on how these individusds would 
react to the test experience, the same questionnaire should also be 
administered to the control group. 

3. Statistical Analysis 

Statistical comparisons of the groups on both the test and 
questionnaire items should be done with a standard "F' test. While not 
a critical component of this study, simple reliability analysis could be 
run on the test items to see if there are any particular items that had 
flaws, or that have great performance differences between delivery 
media. This reliability analysis will be done by manually scanning the 
data on the individual items. 

B. Co«t Of Hie Evaluation 

Eighty individuals, each paid $80. will make the baseline cost for the 
evaluation $6,400. In addition, the cost of the evaluation will include 
the translation of the simulation test items to actual vehicles, the 
"rental" of the vehicles, and the personnel time of BAR staff to set-up 
the test problems on the vehicles and administer the test to the 40 
participants in the control group. 

C. Summaiy 

It should be noted in conclusion that the purpose of the evaluation is 
to And out if any significant differences in performance exist between 
the experimental group of mechanics tested through simulation and 
those in the control group tested on actual vehicles. Regardless of the 
outcome of the statistical analysis of the data, the results of this 
evaluation will be logically analyzed in relationship to the initial 
purpose of the study. This process will include a discussion of the size 
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and implications of any differences that may be detected between 
groups, and a comparison of the test scores of the groups with the 
questionnaire data. For example, if there is no performance 
difference, or if the control group tested on vehicles does only slightly 
better, then it would seem that the simulation testing does perform its 
function of determining the knowledge, skills and abilities of smog 
check mechanics. Markedly different results will require additional 
analysis of sample selection, test construction and administration, and 
other factors that could have a confounding effect. 
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Appendix B 
Simulation Test Documents 



Software and Hardware used in Producing the BAR Demo Project 

MacroMedia Director 3.1 

MacroMedia Model 

Adobe Photoshop 2.0.1 

Nikon LS-3510AF 35mm Slide Scanner 

Claris MacDraw Pro 1.0 

Microsoft Excel 3.0 

Microsoft Word 5.1 

CoSa PACo 

VideoLoglc MIC System II 
SuperMac Video Spigot 
Premier 2.0 
Sound Edit Pro 
Troll Touch 
Pioneer XObject 
Trig XObject 
MIC XObject 

Macromedia MacRecorder 

Hardware Configuration for BAR Demo Project 

Apple Macintosh Ilci, 8meg RAM, 120 meg hard drive 

SuperMac SuperMatch 17 multimode monitor with Troll Technoligies 

Touch screen 
VideoLoglc DVA4000 video and graphics board 
Pioneer 4400 Video Disc Player 
Sony Headphones 
Vaniman Custom Audio Mixer 
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TEST ITEM SUMMARY 



VISUAL INSPECTION 

1. Test Item 1: 

89 Toyota Tercel 

Engine Configuration, required ECS, use of reference material in absence of 
underhobd label. 

2. Test Item 2: 

89 Toyota Tercel 
Visual inspection of TAC 

3. Test Item 3: 

90 Buick Reatta 

Visual Identification of EGR type and visual inspection of EGR system. 
FUNCTIONAL INSPECTION 

4. Test Item 4: 
92 Ford Taurus 
Functional Timing Test 

5. Test Item 5: 

89 Toyota Tercel 
EGR Functional Test 

DIAGNOSTIC 

6. Test Item 6: 
Various Components 

Computer-related components and categorization by "sensor/actuator." 

7. Test Item 7: 
92 Ford Taurus 

Diagnostis of high HC/CO problem. 

8. Test Item 8: 
92 Ford Taurus 

Repair Order entries for parts and labor to repair problems found in Test Item 7. 

9. Test Item 9: 

90 Buick Reatta 

Diagnostic Trouble Code. Tech should be able to use vehicle's On Board 
Diagnostic (OBD) system and identify definition of fault code. 
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SMOG CHECK TECHNICIAN CHECKLIST 

Answer Key 

NABIE: DRIVERS LIC#: • 



BAR UC. #: DATTIMB FHONB:. 



ITEM # 1 (1989 TOYOTA) 

9ue8t#Points 

1 1 Vm NR:_JT2£L31D4K0425393_ 

2 1 UCENSE NR: OS 

3 1 MODEL: TERCEL 



4 1 ENGINE SIZE! 1.5 or ISOO or 88.9 U NIT OF BIEASURE: LorCCorCID 

5 1 CERTIFICATION STATUS (CHECK ONE BELOW): 

6 1 _X_CALIFORNIA EBAISSION CERTIFIED 

^_FEDERAL EBflSSlON CERTIFIED 
B AR CERTIFED 
G REYBiARKET 

7 1 WHAT TYPE OF TRANSBHSSION DOES THIS VEHICLB HAVE? 

8 1 a«ANUALTRANS:_X_ AUTOBIATIC TRANS: 

WHICH OF THESE DEVICES ARE REgUIRED ON THIS VEHICLE? 
(CHECK ALL APPROPRIATE ITEMS): 

Item# Points 

8 1 PCV:_y_ 13 1 FUELINJECTION:_jy_ 

9 1 AIS:_y_ 14 1 TAC:_y__ 

10 1 FUEL RESTRICTOR:_y. 15 1 EVAP SYSTEM: Y_ 

11 1 TWC:__y_ 16 1 EGR:_y_ 

12 1 SPARK ADVANCE:_y_ 17 1 CCO:__y_ 
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ITEM #2 (1989 TOYOTA) 

Quett#Points 

18 1 VISUiULINSPBCTIONOPTAC: (CHECK ONE) 

PASS ( ) BUSSING ( X ) MODIFIED ( ) 

DISCONNECTED ( X ) DEFECTIVE ( ) NOTAPPUGABLE ( ) 

ITEM #3 (1990 BUICK RIATTA) 

19 1 VISUAL INSPECTION OF EGR: (CHECK ONE) 

PASS ( X ) mSSING ( ) MODIFIED ( ) 

DISCONNECTED ( ) DEFECTIVE ( ) NOT APPLICABLE ( ) 

20 1 TYPE OF EGR VALVE: (CHECK ONE BELOW): 

CONVENTIONAL: 

BACK PRESSURE: 



ELECTRONIC/DIGITAL: X 
NOT APPUCABLE: 

ITEM # 4 (1992 FORD TAURUS) 

21 2 FUNCTIONAL THONG TEST: (ENTER TOUR FINDINGS) 

DEGREES: 10 BTDC ( X ) ATDC ( ) 

ITEM #5 (1989 TOYOTA TERCEL) 

X X DOES THIS VEHICLE REQUIRE AN EGR SYSTEM? TES ( ) NO ( 

22 1 IF REQUIRED, WHAT TYPE OF EGR ST8TEM?(CHECK ONE): 

CONVENTlONAL_X_ ELECTRONIC/DIGITA L 
BACK PRESSURE _ X NOT APPUCABLE 

23 1 RESULTS OF FUNCTIONAL EGR SYSTEM TEST (IF REQUIRED): 

PASSED ( X ) FAILED ( ) 
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ITEM # 6 (COMPUTER COMPONENTS) 



9uest#Points 

CHECK THE CORRECT NAME FOR EACH COBIPONENT: 

24 1 COMPONENT A: 

MASS AIR FLOW SENSOR ( ) MAP SENSOR ( X) 

VOLTAGE REGULATOR ( ) COMPUTER PROCESSOR UNIT ( ) 

25 1 COMPONENTS: 

OXYGEN SENSOR ( ) TEMPERATURE SENSOR ( ) 

MASS AIR FLOW SENSOR ( ) THROTTLE POSITION SENSOR ( X) 

26 1 COMPONENT C: 

IDLE AIR CONTROL ( ) EGR POSITION SENSOR (X ) 

CANISTER PURGE ( ) OXYGEN SENSOR ( ) 

27 1 COMPONENT D: 

MASS AIR FLOW SENSOR ( ) KNOCK SENSOR ( ) 

OXYGEN SENSOR ( X ) CRANKSHAFT THONG SENSOR ( ) 

28 1 COMPONENT E: 

FUEL INJECTOR ( X) COOLANT TEBIP. SENSOR ( ) 

MAP SENSOR ( ) OXYGEN SENSOR ( ) 



CHECK THE APPROPRIATE CLASSIFICATION FOR EACH COBAPONENT: 

29 1 BCAP: ACTUATOR ( ) SENSOR ( X) N/A ( ) 

30 1 BiASS AIR FLOW: ACTUATOR ( ) SENSOR ( X ) N/A ( ) 

31 1 FUEL INJECTOR: ACTUATOR (X) SENSOR ( ) N/A ( ) 

32 1 COBIPUTER PROCESSING UNIT: ACTUATOR ( ) SENSOR ( ) N/A (X) 

33 1 CANISTER PURGE SOLENOID: ACTUATOR (X) SENSOR ( ) N/A ( ) 
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ITEM # 7 (1992 FORD TAURUS) 

9uett#Poi]its 

34 1 RECORD THE FIRING ORDER OP THIS VEHICLE IN THE SPACE BELOW: 

FIRING ORDER: 1 -- 4 --- 2_— 5_— _3 — 6_ 

35 6 WHICH OF THE ENGINE'S CYLINDER OR CYLINDERS IS/ARE CAUSING 

THE TAILPIPE EMISSIONS TO BE SO HIGH? 

ANSWER: 2816 HANDS-ON 4Si5 INTERACTIVE__ 



FOR EACH CYLINDER THAT YOU INDICATED ABOVE, IDENTIFY WHAT IS 
THE MOST LIKELY CAUSE: (BIATCH CAUSE TO CYLINDER NUMBER) 

CAUSES: CYLINDERS 

guesttPoints HO INT 

A. BURNED EXHAUST VALVE 36 1 1 

37 1 2 C 

B. EXCESSIVE FUEL IN THE CRANKCASE 38 1 3 

39 1 4 C 

C SPARK PLUG SHORTED 40 1 5 D 

D DISCONNECTED OR OPEN 41 1 6 D 

SPARKPLUG WIRE 

42 1 WHAT IS THE REPAIR COST UBIIT FOR THIS VEHICLE? $ 300 
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ITEM # 8 (1992 FORD TAURUS) 



BASED UPON TOUR EXAMINATION AND RECOMMENDED REPAIRS ON 
THE FORD TAURUS, PLEASE COMPLETE THE PARTS AND LABOR 
SECTIONS OP THE ATTACHED REPAIR ORDER. 



YOU ARE NOT R EQUIRED TO MAKE ANT ENTRIES REGARDING £Q8X8. 



STAT AUTO REPAIR 

10240 Systems Pkwy 
Pomona. CA 90000-0000 
(714)929-0000 
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BEST COPY AVAILABLE 



T 



ITEM # 9 (1990 BUICK REATTA) 



QuestfPoints 

ASSUME THE "CHECK ENGINE^' UGHT WAS ON CONTINUOUSLY DURING 
YOUR ROUTINE SMOG CHECK. 

DOWNLOAD THE TROUBLE CODES FOR THIS VEHICLE TO DETERMINE IP 
ANY WERE SET. 

IP ANY TROUm^ CODES WERE POUND, ENTER THEM BELOW : 



43 1 TROUBLE CODE(S):_E022H OR b333c K0_E021c OR hll2c 
INTERACnVE_ 



W THERE WAS AN EBHSSION RELATED TROUBLE CODE(S). WHAT WAS 
THE MEANING OP THAT CODE? 



44 1 ^COOLANT SENSOR TEBSP HIGH 

X TPS SIGNAL VOLTAGE LOW 

^LOSSOPSIRDATA 

^OPEN 02 SENSOR CIRCUIT 
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Manual List for BAR Demo Test Station 

1 . 1989 Toyota Tercel Repair Manual 

2. 1992 Fbrd Car /Truck Service Manual, Powertrain Control/Emissions 
Diagnosis 

3. 1990 Buick Service Manual-Riviera/Reatta 

4. State of California Licensed SMOG Check Inspection Manual, Revision 2 

5. 1992 Edition Emission Control System Application, Cascade Automotive 
Resources, Inc. 

6. 1992 Motor Emission Control Manual 12th Edition 

7. 1989-92 Motor Chrysler /Ford Auto Engine Ttme-Up & Electronics Manual, 
8th Edition 

8. 1989-92 Motor General Motors Auto Engine Tune-Up & Electronics 
Manual, 8th Edition 

9. 1992 Mitchell Engine Performance Service & Repair, Domestic Cars 

10. 1990 Mitchell Engine Performance Service & Repair, Domestic Cars 

11. 1989 Mitchell Engine Performance Service & Repair, Imported Cars, Light 
Trucks & Vans 
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Appendix C 
Survey for Mechanics 



SURVEY FOR MECHANICS 

NAME: 



BAR DEMONSTRATION PROJECT 

DATE OF BIRTH: 



see. SECURITY NUMBER: ^LOCATION: 



DATE OF TEST: ^YEARS AS A PROFESSIONAL MECHANIC 

YEARS AS A CERTIFIED SMOG CHECK MECHANIC 

We are very interested in how you feel about the testing experience that you 
have Just completed. To the right of the statements listed below are the 
numbers 1-5. #1 means that you strongly agree with the statement next to 
it. #2 means you agree with the statement; #3 means you're undecided : #4 
means you disagree ! and #5 means you strongly disagree with the statement. 

Please think about each statement and circle the number to the right that 
best describes your personal feelings about that statement. 

Is strongly agriee 2s agree 3* undecided 4k disagree 5* strongly disagree 

a This test gave me a fair chance to show my 

skills as a "Smog Check" mechanic. 1 2 3 4 5 

b. This test was very easy for me. 1 2 3 4 5 

c. Most of the questions on this test were very much 

like the things that I really do on my Job. 1 2 3 4 5 

d. This test gave me a fair chance to show my 

knowledge of the "Smog Check" program. 1 2 3 4 5 

e. Most mechanics that are certified for the "Smog 

Check" program would say that this was a hard test. 1 2 3 4 5 

f. I think that all the "Smog Check" testing should be 

done Just like the test that I Just took. 1 2 3 4 5 

g. The best kind of "Smog Check" test is the method 

using a written test, pencil and answer sheets. 1 2 3 4 5 

h. The way that the test I Just took was given to me 

was very confusing. 1 2 3 4 5 

i. I enjoyed taking a test using this method. 1 2 3 4 5 

J. Most of the things that this test asked me about I 

don't usually do on my Job. 1 2 3 4 5 
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Appendix D 
Sample Selection Doctiments 



Sisteof CsBoniiti 

MEMORANDUM 



Departmsnt of Conaimar Affairs 
Date: K^arch9,1993 



To: Larry Harty, Director 

Instructional Tachnology Center 

California State Polytechnic University, Pomona 



From: Bureau of Automotive Repair 
George Adelsperger, Manager 
Standards and Training Branch 



Subject: RANDOM SELECTION METHOD 

in mid-October, 1993 you were provided with a list of 1500 Smog Check technicians. 
They had been sent a letter from me asking them to contact you if interested in 
participating in the study. All of the letters were mailed in the same week. No record of 
the names has been kept by BAR. 

The list was generated by using a computer program to randomly select 1500 names 
from a pool of about 8,600 names. The pool consisted of all of the Unlimited (EU) 
licensees whose licenses were "clear" in the five counties listed below. 

A "clear" license is one that is not expired, and has no pending enforcement actions. 
An Unlimited (EU) license allows the technician to perform inspections, diagnoses and 
repairs on all model years of vehicles. The Limited license category allows such 
actions on only 1979 and older vehicles, and would not have been appropriate for this 
study. The quantity of Limited licensees is very small. 

The counties used were Los Angeles, Orange, Riverside, San Bernardino, and 
Ventura. 
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fi^i -it ^4..tn.<..t • iixvt MO jiwuMg* ttm<,.e.s ,eiKf» Pate ^ ^^^^ ^vernoj 

Constunur 



BUREAU OF AUTOMOTIVE REPAIR 

10240 SYSTEMS PARXWaV. SACRAMENTO. CA 95827 



9^ 

^onstunur ^ ^ j .<r' 




I.D.* ED042501 



GARCZLAZO MIGUEL A19GEL 
P O BOX 7081 

ORANGE . CA 92613 



We wish to invite you to help ua develop a new and more meaningful way 
of licensinc Smog Check technicians in the -future. 

The Bureau of Automotive Repair (BAR) has been working with the 
California State University system to develop a simulation of a 
handa-on examination by using the latest in computer and television 
technology. In order to show the usefulness of this technology, we 
need to compare the performance of two groups of technicians like you. 

Members of one randomly selected group will be asked to perform 
typical Smog Check tasks (inspection, diagnosis, repair) on actual 
cars. This would be similar to the hands-on exercises you may have 
been asked to do by a Quality Assurance Inspector. Members of another 
group will be asked to perform- computer simulations of the same tasks. 
This experience would be a lot like using a Smog Check analyzer. 

The study will be performed by the California State Polytechnic 
University (Cal-Poly), Pomona. All performance information will be 
strictly confidential and controlled by Cal Poly, and will not be 
given to BAR. 

Your participation should take less than a half day and will have 
absolutely no effect upon your license. You will receive $80.00 
compensation for your efforts. 

This study is scheduled for January 18th to February 12t'h, 1993. We 
will be needing only 80 participants so your prompt reply is 
encouraged. 

If you are interested, please call Cal Poly at (714) 869-3922 as soon 
as possible. Inform the person who answers that you wish tc sign up 
for the BAR Demonstration Project. 

We hope you can give us a hand. 



Sincerely, 



George AdeLsperger^ Manager » 
Standards and Training Branch - 
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3801 West Temple Avenue 
Pomona, California 91768-4049 
Telephone (909) 869-3920 



Instructional Technology Center 



Hellol 



Thank you for your quick response to the BAR letter of invitation. 

This is to inform you that your name has been entered into a pool of approximately three 
hundred Smog Check technicians like yourself. One hundred finalists will be randomly 
selected from the pool on December 10th. You will be informed soon thereafter if you 
have or have not been selected. 

If selected, you will be asked to appear at a time and date at Cal Poly Pomona or on the 
campus of Citrus College in Glendora. If possible, we will accommodate your morning or 
afternoon scheduling needs. The site choice is part of our random selection. 

If you continue to be Interested, please complete the following form and 
return this letter in the enclosed envelope. 



I would definitely be available to participate in the BAR Smog Check 
Study, at either location, during the period of Jan 18 thru Feb. 12, 
1993. I understand that if I am chosen for this project, my 
participation should take less than half a day and that I will receive 
$80.00 in compensation for my time. 



Circle one to indicate preference: 8-I2 AM 1-5 PM Either 

Mailing address: 

Telephone Number:^ > ^(Day) _(Eve) 

BAR License No: EU Social Sec. No: - 

Please print your name: . 

Signature: Date: , 



BAR Smog Check Study 



CONFIRMATION OF INTENT TO PARTICIPATE 
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Agriculture • Arts • Business Administration • Engineering • EnvironmenUl Design • Science 
School of Education • School of Hotel and Restaurant Management 
Member of Hie Califogi^Utt Univmity 




3801 West Temple Avenue 
Pomona, California 91768-4049 
Telephone (909) 869-3920 



Instructional Technology Center 



January 4, 1993 



Congratulations! 



You have been selected to participate in the Bureau of Automotive Repair Smog Check 
Study. 

We ask that you appear promptly at 1 :00 pm on February 8. 1993. at the campus of 
Califomia State Polytechnic University, Pomona (map enclosed). 

We look forward to seeing you and we appreciate your willingness to help us in this imp 
endeavor. 

Best Wishes, 

The BAR Smog Check Study Staff 
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3801 West Temple Avenue 
Pomona, California 91768-4049 
Telephone (909) 869-3920 



instructional Technology Center 



January 4, 1 993 



Congratulations! 

You have been selected to participate in the Bureau of Automotive Repair Smoq Check 
Study. 

We ask that you appear promptly on February 9,1993 at 1 :30 pm, at the Citrus College 
campus (map enclosed). 

You will be looking for the Automotive Technology Department at Citrus College. Use the 
Barranca St. entrance to the parking lot just North of the Athletic field. 

PLEASE CALL (909) 869-3922 TO CONFIRM YOUR INTENTIONS TO PARTICIPATE ON 
THIS DATE AND TIME. 

We look forward to seeing you and appreciate your willingness to help us in this important 
endeavor. 



Best Wishes, 

The BAR Smog Check Study Staff 
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3801 West Temple Avenue 
Pomona, California 91768-4049 
Telephone (909) 869-3920 



Instructional Technology Center 



January 14, 1993 



We're Sorryl 



We appreciate your willingness to help us with the Bureau of Automotive Repair Smog Check 
Study, but regret to inform you that, due to the large number of applicants, your name did not 
come up in the random selection process. 

We will keep your name on file in case we need emergency substitutes or do a follow-up 
program. Thank you again for your response. 



Best Wishes for the New Year, 
The BAR Smog Check Study Staff 
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HOI 


32 


2 


1 


2 


o 


o 
c 


1 
1 


o 
c 


1 


R 
w 


R 
w 


1 


R 
w 


H02 


23 


4 


1 


1 


q 
o 


A 
*» 


o 


A 
•» 


o 


R 
w 


4 


o 
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t» 


H03 


29 


12 
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c 
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c 


o 


q 
o 


R 
w 
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o 
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w 


H04 
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1 
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•» 
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•» 
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c 


A 
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O 
t» 


o 
c 
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•» 
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2 
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o 
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w 
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w 
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1 
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w 
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37 


20 


12 


1 


1 


1 
1 


1 

1 


O 


o 
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w 


R 
w 
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1 
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w 
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30 


5 


5 


2 


2 
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1 
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2 
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2 
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q 
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R 
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3 


5 


1 


2 
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47 


32 


20 


2 


2 


3 


2 


1 


1 
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2 


1 
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38 


6 
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1 


1 


1 


1 


5 


1 


5 


5 


1 


5 
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28 


0.5 
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2 


2 


2 


3 


3 
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2 
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40 
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o 
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1 
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w 
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36 


19 
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2 
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o 


q 


o 

€, 


o 


q 
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q 

w 


P 


q 

w 


SMS 


38 


15 


7 


3 
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O 


o 


A 


q 


•t 
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w 


4 


q 


4 


SM4 


44 


24 


24 


4 


q 

O 


o 


4 
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€, 


A 


1 

1 


1 




4 


SMS 


46 


27 


16 


1 


o 


i 
1 


1 


O 


i 


q 


q 

w 


i 


R 


SM6 


55 


4 


2 


1 


o 


1 


i 
1 


o 
c 


i 


q 

w 


R 


i 


1 
1 


SM7 


35 


8 


8 


1 


o 
c 


1 
1 


1 


o 


1 


2 

Cm 


3 


1 


3 


SMS 


28 


10 


2 


3 


1 

1 


1 


2 


3 


3 


5 


R 


1 

1 


R 

w 


SM9 


53 


0 


0.33 


1 


o 

€, 


1 


1 


q 

w 


1 
1 


R 


q 


1 


R 


SM10 


28 


1 1 


7 


1 


1 
1 


1 


1 


R 


1 


R 
w 


A 


1 


R 
w 


SM11 


37 


22 


3 


1 


q 

w 


1 


1 


3 


3 


4 


4 


2 

Cm 


R 
w 


SM12 


52 


35 


15 


2 


A 
*» 


o 


o 

€, 


4 


O 


4 


4 


o 

Cm 


q 


SM13 


26 


6.5 


3.5 


1 


o 


i 
1 


1 
1 


4 


o 


3 


4 


2 


R 


SM14 


31 


5 


3.5 


2 


q 


q 


1 


q 


1 


q 


q 




O 
t» 


SM15 


29 


5 


4 


2 


o 


q 


o 

€, 


o 

€, 


4 


1 


q 


o 


R 


SM16 


62 


30 


7 


2 




o 


O 


O 


q 


4 


o 


o 


4 


SM17 


37 


20 


7 


1 




i 
1 


1 


1 


i 
1 


■ R 


A 


1 

1 


R 


SM18 


22 


5 


5 


2 


3 


1 

1 


2 


3 


2 


2 


2 


2 


4 


SM19 


39 


16 


2 


1 


2 


1 
1 


1 


2 


2 


3 


R 


1 


R 

yj 


SM20 


31 


12 


6 


1 


q 

w 


1 
1 


1 


2 

Cm 


3 


4 


4 


2 


4 


SM21 


26 


7 


1 


2 


2 


3 


2 


3 


2 


R 

W 


4 


2 


4 


SM22 


42 


22 


19 


2 


3 


4 


2 


3 


3 


R 


3 


2 


2 


SM23 


52 


15 


5 


1 


2 


2 

Cm 


1 


2 

Cm 


2 

Cm 


4 


4 


1 


2 


SM24 


38 


21 


21 


2 


3 


4 


1 


3 


2 


4 


3 


1 


3 


SM25 




12 


12 


2 


3 


1 


2 

Cm 


2 

Cm 


2 


R 


2 


1 


4 


SM26 


46 


23 


12 


2 


3 


q 

w 


2 


A 
•t 


3 


4 


4 


2 

Cm 


3 


SM27 


35 


15 


12 


3 


3 


2 


3 


1 


3 


R 
w 


3 


3 


4 


SM28 


34 


15 


6 


4 


2 

Cm 


1 


4 


4 


2 

Cm 


4 


4 


1 


1 


SM29 


36 


16 


8 


2 


1 




2 


q 

w 


3 


A 
•t 


•t 


1 


R 


SM30 


38 


1 1 


5 


2 


3 


4 


2 


4 


4 


4 


R 

yj 


1 


4 


SM31 


31 


9 


9 


1 


2 

Cm 


1 
1 


1 

1 


A 
•t 


2 


4 


4 


1 


5 


SM32 


36 


15 


10 


2 


4 


2 


2 

Cm 


4 


2 


3 


4 


2 


9 

*• 


SM33 


35 


1 


1 


2 


2 


1 

1 


1 

1 


3 


2 

Cm 


3 


4 


2 
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W 


SM34 


45 


15 


12 


5 
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BAR Demo Project: System Trials 

Anecdotal Observations 
Made by Sam Hay and Larry Harty 

1 . Several subjects lemaiked that they had "learned something" as a result of taking the 
test. These comments were usually linked to the EGR functional test and the on-board 
computer capabilities of the Buick Reatta. 

2 . The vast majority of the technicians commented that the interactive nature of the exam 
had made the expmence "enjoyable and fun". 

3 . Several technicians who initially expressed Uiat Uiey were intimidated by computers 
seeoMd to lose t!ieir fear when they found the subject matter familiar and the interface 
simple. 

4. Most subjects were quite comfortable with the medium. But those who displayed 
some sli^t anxiety e^peared to quickly adapt and accept the interface when they 
discovered how user Mendly it was. 

5 . Several of the more computer literate participants expressed their surprise at the 
advanced state of the interface and the ability to move freely within and about the 
"environment". 

6. For many of the subjects, determining the location of and navigating to a particular 
part or area of a vehicle appeared to be quite intuitive through Uie use of Uie touch-screen. 
In items such as Uie disconnected TAG, peq>le touched on it and were rewarded wiUi 
discovering that they had obtained addition^ useful information. 

7 . It became quite apparent that the participants were learning to use the touch-screen and 
other interface elements as they answered the fost few questions. We feel that instractional 
and positive reinforcement experiences in the use of Uie interface devices could be designed 
into the initial test items and thus, allow for Uie abreviation of the current "tutorial" section. 

8 . Detailed items such as underiiood labels, should be eiUier very legible or very 
obviously obscured. To provide "almost legible" screens Uiat tantalize the technician tends 
to frustrate and create negative reactions. Tke system intoface should not be suspected as 
a cause of obscurity. 

9 . In many places instructions were repeated unnecessarily causing unneeded delay and 
frustration in the participants. Future versions should require programming features Uiat 
prevent "looping-back" Uirough instmctions unless requested by Uie technician. 

10. A majority of Uie subjects were observed "Uunking ahead" of Uie system. Future 
versions should incorporate resprase times and programming guards to insure Uiat 
technicians are not inhibited or penalized in any way for responding too soon. 

1 1 . Several peqple remariced Uiat they were guessing at component "C" on test item 6 
(Ford EGR position sensor, answer key #26). We are not quite sure why Uiis occurred 
unless it was caused by the two dimensional nature of Uie display. 

1 2. Most participants did not seem to be mentally fatigued after Uie 1 -1/2 to 2 hour testing 
period. 
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1 3 . Most of the participants seemed confident and self assured with their abilities and 
skills as snx>g check technicians. 

14. The age and experience of the participants varied widely fitom young technicians who 
had their smog check license for only a few months to older (Mies who had been licensed 
since die state began the program. 

15. The pTofessicxial working envircMiment of the participants was extremely diverse. 
There were educators, dealer mechanics, fleet mechanics, shop owners, service managers 
and those who did general mechanics as well as those who only did smog checks. 

One perscMi had only worked on Jaguars c(»npleted the test and performed very well. 

16. The ethnic mix of the persons in the test pool was quite diverse. 

17. All participants were male. 

1 8. It was interesting to note that the interactive group scored ccmsistenUy better tiian die 
hands-on group in answer key items 8 through 17. Since these questions were essentially 
pencil and paper type items we did not expect to see a difference in the two groups. 
Perhaps, the novel, interactive nature of the medium was more engaging and thereby 
resulted in better performances. 
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Anecdotal Notes of Hands-On Exam 

by niilip DeLeon, State of California 
Department of Consumer Affairs, Bureau of Automotive Repair 



1. Of the 38 randomly selected technicians, four of them were Certified Smog Check 
Instructors, one worked for a Chevrolet dealership, two for Cadillac, one for Honda, one 
for Lexus, one woiked for the BAR referee, and another as a representative for BAR. Of 
the 38 technicians, one was female. 

2. Several of the technicians expressed that they were pleased BAR was planning to 
implement some type of hands-on examination. They felt that this would improve the 
quality of service that consumers would receive from a shop. 

3. One primary difference between the hands-on and the interactive versions was 
that the hands-on ^up could remove parts from the engine (e.g., spark plugs) to confirm 
suspect problems. 

4. Some were confused in that different reference manuals supplied conflicting 
information (as in the real worM). This was particularly dependent as to which manual 
(e.g.. Motor Manual vs Mitchell Manual) they referenced. 

5. Some had difficulty determining the actual readings (e.g., timing marks) due to 
poor vision or poor visibility. Others had difficulty interpreting the letters or digits (e.g., 
mistaking a "b" for a "6" or an "H" for a "4" when reading the trouble codes). I think 
that the interactive exam may yield better control over these types of extemporaneous 
conditions. 

6. A couple of individuals said they would have preferred the word "retrieve" 
instead of "download" when it came to downloading the trouble codes firom the Reatta. 

7. Upon completion of the exam, many reported that they learned at least one thing 
that they did not {ueviously know. 

8. A few technicians are intimidated by using reference manuals when they ran into 
an area unfamiliar to them. 

9. A good number of technicians did not always follow manufacture's instructions 
when performing tasks. Sometimes they would opt to perform a more generic test 

10. Some used "creative" methods to perform certain tasks such as looking at the 
arrangement of spark plug wires on the distributor cap as opposed to looking up the 
answer in the reference manuals when they didn't know the answer outright. Other 
things included looking at the doorpost label for a VIN instead of looking at the lower left 
hand comer of the windshield (the preferred noethod). 

11. A similar adaptation of this interactive and multimedia technology could be used 
as an educational tool to compliment the interactive exam. The information recorded 
from the exam results could be used as a means to strengthen areas identiHed as common 
weak areas for technicians. 
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